Skip to content

statistics: reduce TopN/buckets collected for non-predicate columns | tidb-test=pr/2767#69667

Closed
terry1purcell wants to merge 6 commits into
pingcap:masterfrom
terry1purcell:predmin
Closed

statistics: reduce TopN/buckets collected for non-predicate columns | tidb-test=pr/2767#69667
terry1purcell wants to merge 6 commits into
pingcap:masterfrom
terry1purcell:predmin

Conversation

@terry1purcell

@terry1purcell terry1purcell commented Jul 4, 2026

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Issue Number: close #69668

Problem Summary:

ANALYZE collects the same number of TopN values and histogram buckets for every column (default 100/256), so non-predicate columns — which never drive plan selection — cost as much to collect, store, and cache as predicate columns.

What changed and how does it work?

Add the global variable tidb_analyze_non_predicate_column_ratio (default 0.1, range [0,1]). During ANALYZE v2, a column that is not a predicate column collects only ratio × the default TopN and bucket numbers (100/256; buckets floored at 1).

The reduction only applies when the user relied on the defaults. TopN/bucket numbers explicitly requested by the user — WITH n TOPN / WITH n BUCKETS in the statement, or persisted analyze options — are always honored and never reduced, and each option is handled independently (e.g. WITH 15 TOPN keeps TopN at 15 for every column while the bucket number, left at the default, is still reduced for non-predicate columns).

Columns that keep the full numbers:

  • predicate columns recorded in mysql.column_stats_usage, when any exist for the table;
  • otherwise the handle column and the first column of each index (most likely future predicate columns);
  • columns explicitly specified in ANALYZE TABLE ... COLUMNS.

Index statistics are never reduced. Setting the ratio to 1 disables the reduction.

Implementation: the full-stats column set and the ratio are decided at plan-build time (PlanBuilder.getFullStatsColsAndRatio) and carried on AnalyzeColumnsTask to the analyze executor, which passes the per-column ratio into BuildHistAndTopN. The reduction is applied inside BuildHistAndTopN, where the default-vs-user-specified distinction already exists, so reduced numbers are still treated as default-derived: TopN pruning and the NDV-adaptive bucket shrink continue to apply. Consequently small/low-NDV columns produce the same statistics as before, and the reduction only takes effect where the defaults actually bind (high-NDV or heavily skewed columns). Auto-analyze plans through the same path, so it inherits the setting.

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No need to test
    • I checked and no code files have been changed.

New test TestAnalyzeNonPredicateColumnRatio covers the predicate-column case, the no-usage-info fallback, explicitly specified columns, user-specified WITH n TOPN / WITH n BUCKETS being honored (together and independently), and disabling via ratio = 1. No existing test needed to pin the ratio and no integration test results changed: because reduced columns keep TopN pruning and adaptive bucket sizing, existing small-table suites produce identical statistics.

Side effects

  • Performance regression: Consumes more CPU
  • Performance regression: Consumes more Memory
  • Breaking backward compatibility

With the 0.1 default, the first ANALYZE of a never-queried table collects reduced histograms for columns other than the handle and first index columns, which can coarsen estimates until predicate-column usage is recorded and the table is re-analyzed. This only affects columns where the default TopN/bucket numbers actually bind (high NDV or heavy skew); low-NDV columns produce the same statistics as before. Explicitly requesting TopN/bucket numbers in ANALYZE opts the statement out of the reduction, and tidb_analyze_non_predicate_column_ratio = 1 restores the previous behavior globally.

Documentation

  • Affects user behaviors
  • Contains syntax changes
  • Contains variable changes
  • Contains experimental features
  • Changes MySQL compatibility

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

Add the global variable `tidb_analyze_non_predicate_column_ratio` (default 0.1). When ANALYZE runs with the default TopN and bucket numbers, it now collects a reduced number of TopN values and histogram buckets for columns that are not used in query predicates, lowering analyze cost and statistics size. TopN and bucket numbers explicitly specified in the ANALYZE statement or persisted analyze options are always honored. Set the variable to 1 to restore the previous behavior.

Summary by CodeRabbit

  • New Features

    • Added a new global setting to control how much ANALYZE collects for non-predicate columns.
    • Column stats collection now preserves full detail for selected columns while reducing collection for others when the setting is lowered.
  • Bug Fixes

    • Improved ANALYZE behavior so TopN and histogram bucket counts are handled more consistently across predicate and non-predicate columns.
    • User-specified TopN and bucket settings continue to be honored as expected.
  • Tests

    • Added coverage for the new ratio setting and its impact on statistics output.

Add the global variable tidb_analyze_non_predicate_column_ratio (default
0.1, range [0,1]). During ANALYZE v2, columns that are not predicate
columns collect only ratio times the configured TopN and bucket numbers
(buckets floored at 1). Columns that keep the configured numbers are:

- predicate columns recorded in mysql.column_stats_usage, when any exist;
- otherwise the handle column and the first column of each index;
- columns explicitly specified in ANALYZE TABLE ... COLUMNS.

Index statistics are never reduced. The full-stats column set is decided
at plan-build time and carried on AnalyzeColumnsTask to the analyze
executor, so auto-analyze picks it up as well. Setting the ratio to 1
disables the reduction.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@ti-chi-bot ti-chi-bot Bot added do-not-merge/needs-linked-issue do-not-merge/needs-tests-checked release-note-none Denotes a PR that doesn't merit a release note. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. component/statistics sig/planner SIG: Planner and removed do-not-merge/needs-tests-checked labels Jul 4, 2026
@ti-chi-bot

ti-chi-bot Bot commented Jul 4, 2026

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign terry1purcell for approval. For more information see the Code Review Process.
Please ensure that each of them provides their approval before proceeding.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@coderabbitai

coderabbitai Bot commented Jul 4, 2026

Copy link
Copy Markdown

Review Change Stack

📝 Walkthrough

Walkthrough

This PR adds a new global system variable tidb_analyze_non_predicate_column_ratio (default 0.1) that scales down TopN and histogram bucket counts collected during ANALYZE for non-predicate columns. Planner logic determines which columns retain full stats, threading this through the executor and BuildHistAndTopN, with accompanying test coverage.

Changes

Non-predicate column ratio feature

Layer / File(s) Summary
New system variable definition
pkg/sessionctx/vardef/tidb_vars.go, pkg/sessionctx/variable/sysvar.go
Adds TiDBAnalyzeNonPredicateColumnRatio constant, DefTiDBAnalyzeNonPredicateColumnRatio default (0.1), atomic global holder, and sysvar registration with GetGlobal/SetGlobal (range 0..1).
Planner computation of full-stats columns and ratio
pkg/planner/core/common_plans.go, pkg/planner/core/planbuilder.go
AnalyzeColumnsTask gains FullStatsCols/NonPredicateColRatio fields; getPredicateColumns gains a warnEmpty flag; new getFullStatsColsAndRatio helper derives full-stats columns and effective ratio; buildAnalyzeFullSamplingTask wires these into each task.
Executor scaling of TopN/bucket counts
pkg/executor/analyze_col.go, pkg/executor/builder.go, pkg/executor/analyze_col_sampling.go
AnalyzeColumnsExec adds fullStatsCols/nonPredicateColRatio fields populated from the task; subBuildWorker computes a per-column ratio and passes it along with numBuckets into BuildHistAndTopN.
BuildHistAndTopN ratio-based pruning
pkg/statistics/builder.go
BuildHistAndTopN gains a nonPredicateColRatio parameter, scales numTopN/numBuckets when ratio < 1 and inputs are still default, clamping buckets to at least 1.
Test updates and new ratio test
pkg/executor/test/analyzetest/columns/*, pkg/statistics/builder_test.go, pkg/statistics/sample_test.go, pkg/statistics/statistics_test.go
Adds TestAnalyzeNonPredicateColumnRatio covering multiple ANALYZE scenarios; existing BuildHistAndTopN call sites updated with the new trailing argument; Bazel shard count bumped.

Estimated code review effort: 3 (Moderate) | ~30 minutes

Sequence Diagram(s)

sequenceDiagram
  participant PlanBuilder
  participant AnalyzeColumnsTask
  participant Builder
  participant AnalyzeColumnsExec
  participant BuildHistAndTopN

  PlanBuilder->>PlanBuilder: getFullStatsColsAndRatio()
  PlanBuilder->>AnalyzeColumnsTask: set FullStatsCols, NonPredicateColRatio
  Builder->>AnalyzeColumnsTask: read FullStatsCols, NonPredicateColRatio
  Builder->>AnalyzeColumnsExec: set fullStatsCols, nonPredicateColRatio
  AnalyzeColumnsExec->>AnalyzeColumnsExec: compute per-column nonPredicateColRatio
  AnalyzeColumnsExec->>BuildHistAndTopN: BuildHistAndTopN(numBuckets, nonPredicateColRatio)
  BuildHistAndTopN->>BuildHistAndTopN: scale numTopN/numBuckets when ratio < 1
Loading

Suggested reviewers: YangKeao, 0xPoe, AilinKid

Poem

A rabbit hops through columns wide, 🐇
Predicate ones keep their stride,
Others shrink their TopN hoard,
Ten percent is all they're stored,
Set the ratio, watch stats glide! ✨

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Linked Issues check ✅ Passed The implementation matches #69668: it adds the ratio, preserves full stats for required columns, and leaves index stats untouched.
Out of Scope Changes check ✅ Passed No unrelated code changes are evident; the test sharding tweak appears tied to the added test coverage.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Title check ✅ Passed The title clearly summarizes the main change: reducing TopN/buckets for non-predicate columns.
Description check ✅ Passed The description matches the template with issue number, problem summary, changes, tests, side effects, documentation, and release note.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@ti-chi-bot ti-chi-bot Bot added release-note Denotes a PR that will be considered when it comes time to generate release notes. and removed do-not-merge/needs-linked-issue release-note-none Denotes a PR that doesn't merit a release note. labels Jul 4, 2026
@codecov

codecov Bot commented Jul 4, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 69.44444% with 22 lines in your changes missing coverage. Please review.
✅ Project coverage is 74.7243%. Comparing base (9f093e6) to head (db55117).
⚠️ Report is 7 commits behind head on master.

⚠️ Current head db55117 differs from pull request most recent head f360d87

Please upload reports for the commit f360d87 to get more accurate results.

Additional details and impacted files
@@               Coverage Diff                @@
##             master     #69667        +/-   ##
================================================
- Coverage   76.3233%   74.7243%   -1.5991%     
================================================
  Files          2041       2059        +18     
  Lines        560726     578746     +18020     
================================================
+ Hits         427965     432464      +4499     
- Misses       131860     145385     +13525     
+ Partials        901        897         -4     
Flag Coverage Δ
integration 42.4129% <69.4444%> (+2.7076%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components Coverage Δ
dumpling 60.4471% <ø> (ø)
parser ∅ <ø> (∅)
br 47.4650% <ø> (-15.2563%) ⬇️
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (4)
pkg/planner/core/planbuilder.go (1)

2479-2519: 📐 Maintainability & Code Quality | 🔵 Trivial

Minor inconsistency: MVIndex/columnar indexes not skipped like in getMustAnalyzedColumns.

getMustAnalyzedColumns (same file, lines 2268-2295) explicitly skips idx.MVIndex || idx.IsColumnarIndex() when picking indexed columns, but this loop does not. Harmless today (the resulting IDs simply won't match anything in colsInfo), but worth aligning for consistency/future-proofing.
[optional_refactor_low_reward]

♻️ Optional consistency fix
 		for _, idx := range tblInfo.Indices {
-			if idx.State != model.StatePublic || len(idx.Columns) == 0 {
+			if idx.State != model.StatePublic || len(idx.Columns) == 0 || idx.MVIndex || idx.IsColumnarIndex() {
 				continue
 			}
 			fullStatsCols[tblInfo.Columns[idx.Columns[0].Offset].ID] = struct{}{}
 		}
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@pkg/planner/core/planbuilder.go` around lines 2479 - 2519, Align
getFullStatsColsAndRatio with getMustAnalyzedColumns by skipping MVIndex and
columnar indexes when collecting index-derived column IDs. Update the fallback
loop over tblInfo.Indices in PlanBuilder.getFullStatsColsAndRatio so it only
adds the first column for public, non-MV, non-columnar indexes, keeping behavior
consistent and future-proof. Use the existing idx.MVIndex and
idx.IsColumnarIndex checks already used elsewhere in planbuilder.go.
pkg/executor/test/analyzetest/options/analyze_saved_options_test.go (1)

40-42: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Missing revert for tidb_analyze_non_predicate_column_ratio global var pin (inconsistent with sibling vars in the same function).

Both TestSavedAnalyzeOptions and TestSavedPartitionAnalyzeOptions snapshot and defer-restore tidb_persist_analyze_options (and TestSavedAnalyzeOptions also does so for tidb_auto_analyze_ratio), but the newly added tidb_analyze_non_predicate_column_ratio = 1 line right next to them is never reverted. Since this is a process-wide atomic global holder (per the PR's variable-definition layer), leaving it at 1 can leak into subsequently-run tests in the same package binary.

♻️ Suggested pattern (apply per occurrence)
-	tk.MustExec("set global tidb_analyze_non_predicate_column_ratio = 1")
+	originalRatio := tk.MustQuery("select @@global.tidb_analyze_non_predicate_column_ratio").Rows()[0][0].(string)
+	defer func() {
+		tk.MustExec(fmt.Sprintf("set global tidb_analyze_non_predicate_column_ratio = %v", originalRatio))
+	}()
+	tk.MustExec("set global tidb_analyze_non_predicate_column_ratio = 1")

As per path instructions for **/*_test.go: "Test files: ... keep test changes minimal and deterministic."

Also applies to: 142-144

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@pkg/executor/test/analyzetest/options/analyze_saved_options_test.go` around
lines 40 - 42, The test setup in TestSavedAnalyzeOptions and the related
saved-options test pins tidb_analyze_non_predicate_column_ratio globally but
never restores it, which can leak state into later tests. Update the test logic
around the existing snapshot/defer pattern used for tidb_persist_analyze_options
and tidb_auto_analyze_ratio so the new global variable is also saved before the
override and restored afterward, keeping the behavior deterministic across the
package binary.

Source: Path instructions

pkg/executor/test/analyzetest/columns/analyze_columns_with_test.go (1)

38-40: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Missing revert for tidb_analyze_non_predicate_column_ratio global var pin.

Each of these six spots sets global tidb_analyze_non_predicate_column_ratio = 1 but never restores the previous value, unlike other global sysvars mutated in the same test bodies elsewhere in this file family (e.g. tidb_persist_analyze_options, tidb_auto_analyze_ratio in analyze_saved_options_test.go, which snapshot the original value and defer a reset). Per the PR context, this variable is backed by a process-wide atomic global holder, not scoped to the per-test mock store/domain, so a leaked value of 1 can silently persist into later tests in the same package binary, masking or skewing the intended default (0.1) reduction behavior for any test that doesn't explicitly re-pin the ratio (the new TestAnalyzeNonPredicateColumnRatio itself correctly reverts via defer, showing the intended convention).

Since none of these six tests actually need a specific ratio (they exist purely to keep pre-existing behavior stable), consider following the same snapshot+defer pattern.

♻️ Suggested pattern (apply per occurrence)
-			tk.MustExec("set global tidb_analyze_non_predicate_column_ratio = 1")
+			originalRatio := tk.MustQuery("select @@global.tidb_analyze_non_predicate_column_ratio").Rows()[0][0].(string)
+			defer func() {
+				tk.MustExec(fmt.Sprintf("set global tidb_analyze_non_predicate_column_ratio = %v", originalRatio))
+			}()
+			tk.MustExec("set global tidb_analyze_non_predicate_column_ratio = 1")

As per path instructions for **/*_test.go: "Test files: ... keep test changes minimal and deterministic."

Also applies to: 109-111, 189-191, 269-271, 398-400, 512-514

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@pkg/executor/test/analyzetest/columns/analyze_columns_with_test.go` around
lines 38 - 40, The `TestAnalyze...` cases in `analyze_columns_with_test.go` pin
`tidb_analyze_non_predicate_column_ratio` with `set global` but never restore
it, which can leak the process-wide value into later tests. Update each
occurrence to snapshot the current value before calling the relevant
`tk.MustExec` setup, then `defer` a reset back to the original setting after the
test body, matching the existing pattern used for other global sysvars in this
test family. Use the `tidb_analyze_non_predicate_column_ratio` setup sites in
the affected test functions as the anchor points for the fix.

Source: Path instructions

pkg/executor/test/analyzetest/analyze_test.go (1)

639-641: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Missing revert for tidb_analyze_non_predicate_column_ratio global var pin.

Same concern as in analyze_columns_with_test.go: these three spots set the global ratio to 1 without capturing/restoring the original value, whereas sibling global vars in these same functions (e.g. tidb_persist_analyze_options) do use snapshot+defer reset. Given this variable is backed by a process-wide atomic global holder, a leaked value can affect later tests in the same package binary.

♻️ Suggested pattern (apply per occurrence)
-			tk.MustExec("set global tidb_analyze_non_predicate_column_ratio = 1")
+			originalRatio := tk.MustQuery("select @@global.tidb_analyze_non_predicate_column_ratio").Rows()[0][0].(string)
+			defer func() {
+				tk.MustExec(fmt.Sprintf("set global tidb_analyze_non_predicate_column_ratio = %v", originalRatio))
+			}()
+			tk.MustExec("set global tidb_analyze_non_predicate_column_ratio = 1")

As per path instructions for **/*_test.go: "Test files: ... keep test changes minimal and deterministic."

Also applies to: 1067-1069, 1164-1166

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@pkg/executor/test/analyzetest/analyze_test.go` around lines 639 - 641, The
test is pinning tidb_analyze_non_predicate_column_ratio globally without
restoring it, which can leak state into later package tests. In each affected
test in analyze_test.go, snapshot the current value before the set in the
relevant test function, use the existing tk.MustExec call to set it to 1, and
add a defer to restore the original value just like the nearby
tidb_persist_analyze_options handling. Refer to the affected test functions
containing the tidb_analyze_non_predicate_column_ratio setup and apply the same
pattern at each occurrence.

Source: Path instructions

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@pkg/executor/test/analyzetest/analyze_test.go`:
- Around line 639-641: The test is pinning
tidb_analyze_non_predicate_column_ratio globally without restoring it, which can
leak state into later package tests. In each affected test in analyze_test.go,
snapshot the current value before the set in the relevant test function, use the
existing tk.MustExec call to set it to 1, and add a defer to restore the
original value just like the nearby tidb_persist_analyze_options handling. Refer
to the affected test functions containing the
tidb_analyze_non_predicate_column_ratio setup and apply the same pattern at each
occurrence.

In `@pkg/executor/test/analyzetest/columns/analyze_columns_with_test.go`:
- Around line 38-40: The `TestAnalyze...` cases in
`analyze_columns_with_test.go` pin `tidb_analyze_non_predicate_column_ratio`
with `set global` but never restore it, which can leak the process-wide value
into later tests. Update each occurrence to snapshot the current value before
calling the relevant `tk.MustExec` setup, then `defer` a reset back to the
original setting after the test body, matching the existing pattern used for
other global sysvars in this test family. Use the
`tidb_analyze_non_predicate_column_ratio` setup sites in the affected test
functions as the anchor points for the fix.

In `@pkg/executor/test/analyzetest/options/analyze_saved_options_test.go`:
- Around line 40-42: The test setup in TestSavedAnalyzeOptions and the related
saved-options test pins tidb_analyze_non_predicate_column_ratio globally but
never restores it, which can leak state into later tests. Update the test logic
around the existing snapshot/defer pattern used for tidb_persist_analyze_options
and tidb_auto_analyze_ratio so the new global variable is also saved before the
override and restored afterward, keeping the behavior deterministic across the
package binary.

In `@pkg/planner/core/planbuilder.go`:
- Around line 2479-2519: Align getFullStatsColsAndRatio with
getMustAnalyzedColumns by skipping MVIndex and columnar indexes when collecting
index-derived column IDs. Update the fallback loop over tblInfo.Indices in
PlanBuilder.getFullStatsColsAndRatio so it only adds the first column for
public, non-MV, non-columnar indexes, keeping behavior consistent and
future-proof. Use the existing idx.MVIndex and idx.IsColumnarIndex checks
already used elsewhere in planbuilder.go.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: 2c5f420b-1a28-4017-9375-735a978ffd25

📥 Commits

Reviewing files that changed from the base of the PR and between b2c55fb and 8306131.

📒 Files selected for processing (14)
  • pkg/executor/analyze_col.go
  • pkg/executor/analyze_col_sampling.go
  • pkg/executor/builder.go
  • pkg/executor/test/analyzetest/analyze_test.go
  • pkg/executor/test/analyzetest/columns/BUILD.bazel
  • pkg/executor/test/analyzetest/columns/analyze_columns_with_test.go
  • pkg/executor/test/analyzetest/options/analyze_saved_options_test.go
  • pkg/planner/cardinality/selectivity_test.go
  • pkg/planner/core/casetest/planstats/plan_stats_test.go
  • pkg/planner/core/common_plans.go
  • pkg/planner/core/planbuilder.go
  • pkg/sessionctx/vardef/tidb_vars.go
  • pkg/sessionctx/variable/sysvar.go
  • pkg/statistics/handle/handletest/handle_test.go

@terry1purcell terry1purcell changed the title statistics: reduce TopN/buckets collected for non-predicate columns statistics: reduce TopN/buckets collected for non-predicate columns | tidb-test=pr/2767 Jul 5, 2026
… ratio

Re-record golden results affected by the tidb_analyze_non_predicate_column_ratio
default of 0.1:

- planner/core/indexjoin: estimate-only shift (51.00 -> 52.50) for a range on
  the second primary-key column, which now collects reduced buckets.
- planner/cardinality/selectivity: show stats_topn/stats_buckets dumps reflect
  the reduced TopN/bucket numbers for non-predicate columns.

Pin the ratio to 1 in two cases whose purpose requires the configured numbers
for index-less tables: executor/kv (asserts mysql.stats_top_n is populated
before DROP STATS) and the TopN-placement tables in
planner/cardinality/selectivity.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR introduces a new global control to reduce ANALYZE v2 statistics collection work for columns that are not (yet) used as predicate columns, lowering analyze cost and the size of stored/cached stats while keeping full stats for predicate/likely-predicate columns and all indexes.

Changes:

  • Add global sysvar tidb_analyze_non_predicate_column_ratio (default 0.1, range [0,1]) and a process-global atomic to hold its value.
  • Decide “full-stats columns” at plan build time and carry the column set + ratio into AnalyzeColumnsTask, then scale per-column TopN/bucket counts during sampling build for non-full columns (buckets floored to at least 1).
  • Update/unit-test coverage and pin the ratio to 1 in existing suites that assert exact stats outputs; adjust related integration test outputs.

Reviewed changes

Copilot reviewed 19 out of 19 changed files in this pull request and generated 15 comments.

Show a summary per file
File Description
tests/integrationtest/t/planner/cardinality/selectivity.test Pins ratio to 1 for specific TopN-placement cases and restores default afterward.
tests/integrationtest/t/executor/kv.test Pins ratio to 1 so mysql.stats_top_n is populated for DROP STATS test flow; restores default.
tests/integrationtest/r/planner/core/indexjoin.result Updates expected estRows output reflecting changed stats behavior.
tests/integrationtest/r/planner/cardinality/selectivity.result Updates expected TopN/bucket result sets under the new default behavior.
tests/integrationtest/r/executor/kv.result Records the added global ratio setting/reset in expected output.
pkg/statistics/handle/handletest/handle_test.go Pins ratio to 1 for a stats-lite init test (needs restoration to avoid leakage).
pkg/sessionctx/variable/sysvar.go Registers new global sysvar with type/limits and atomic backing store integration.
pkg/sessionctx/vardef/tidb_vars.go Adds sysvar name/default and introduces the process-global atomic float64.
pkg/planner/core/planbuilder.go Adds logic to compute full-stats column set + ratio at plan-build time and wires them into analyze tasks.
pkg/planner/core/common_plans.go Extends AnalyzeColumnsTask to carry FullStatsCols and NonPredicateColRatio.
pkg/planner/core/casetest/planstats/plan_stats_test.go Pins ratio to 1 for plan-stats casetest (needs restoration to avoid leakage).
pkg/planner/cardinality/selectivity_test.go Pins ratio to 1 in unit tests that depend on exact TopN/bucket sizes (needs restoration).
pkg/executor/test/analyzetest/options/analyze_saved_options_test.go Pins ratio to 1 for saved-options tests (needs restoration).
pkg/executor/test/analyzetest/columns/BUILD.bazel Adjusts shard_count to accommodate the added unit test workload.
pkg/executor/test/analyzetest/columns/analyze_columns_with_test.go Pins ratio to 1 in existing analyze-column tests and adds TestAnalyzeNonPredicateColumnRatio (pinning needs restoration in multiple places).
pkg/executor/test/analyzetest/analyze_test.go Pins ratio to 1 in analyze tests relying on exact stats output (needs restoration).
pkg/executor/builder.go Passes FullStatsCols and NonPredicateColRatio from plan tasks into the executor.
pkg/executor/analyze_col.go Adds executor fields to hold the full-stats column set and ratio.
pkg/executor/analyze_col_sampling.go Applies per-column scaling of TopN/buckets for non-full columns before calling BuildHistAndTopN.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread pkg/statistics/handle/handletest/handle_test.go Outdated
Comment thread pkg/planner/core/casetest/planstats/plan_stats_test.go Outdated
Comment thread pkg/planner/cardinality/selectivity_test.go Outdated
Comment thread pkg/planner/cardinality/selectivity_test.go Outdated
Comment thread pkg/executor/test/analyzetest/options/analyze_saved_options_test.go Outdated
Comment thread pkg/executor/test/analyzetest/columns/analyze_columns_with_test.go Outdated
Comment thread pkg/executor/test/analyzetest/columns/analyze_columns_with_test.go Outdated
Comment thread pkg/executor/test/analyzetest/columns/analyze_columns_with_test.go Outdated
Comment thread pkg/executor/test/analyzetest/columns/analyze_columns_with_test.go Outdated
Comment thread pkg/executor/test/analyzetest/columns/analyze_columns_with_test.go Outdated
Address review feedback: tests that pin the ratio to 1 now capture the
original global value and restore it with a defer, so the setting cannot
leak into later tests in the same package.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@terry1purcell

Copy link
Copy Markdown
Contributor Author

/retest

@terry1purcell

Copy link
Copy Markdown
Contributor Author

/retest-required

…e columns

The non-predicate column reduction now only applies to the default
TopN/bucket numbers. Numbers explicitly requested by the user (in the
ANALYZE statement or persisted analyze options) are always honored, and
each option is handled independently.

The reduction moves from the analyze executor into BuildHistAndTopN so
that reduced numbers are still recognized as default-derived: TopN
pruning and the NDV-adaptive bucket shrink continue to apply to reduced
columns. As a result, small/low-NDV tables produce the same statistics
as before the reduction, so all previous test pins of
tidb_analyze_non_predicate_column_ratio and the integration test result
changes are reverted.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 2 comments.

Comment thread pkg/executor/analyze_col_sampling.go Outdated
Comment thread pkg/executor/test/analyzetest/columns/analyze_columns_with_test.go
@terry1purcell

Copy link
Copy Markdown
Contributor Author

/retest-required

terry1purcell and others added 2 commits July 6, 2026 00:31
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@ti-chi-bot

ti-chi-bot Bot commented Jul 6, 2026

Copy link
Copy Markdown

@terry1purcell: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
pull-lightning-integration-test f360d87 link true /test pull-lightning-integration-test
pull-integration-realcluster-test-next-gen f360d87 link true /test pull-integration-realcluster-test-next-gen
idc-jenkins-ci-tidb/mysql-test f360d87 link true /test mysql-test
idc-jenkins-ci-tidb/check_dev_2 f360d87 link true /test check-dev2
idc-jenkins-ci-tidb/check_dev f360d87 link true /test check-dev
pull-unit-test-next-gen f360d87 link true /test pull-unit-test-next-gen
idc-jenkins-ci-tidb/unit-test f360d87 link true /test unit-test

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@terry1purcell

Copy link
Copy Markdown
Contributor Author

It was determined that the value of this PR is reduced because the non-predicate columns will not be sync/async loaded.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

component/statistics release-note Denotes a PR that will be considered when it comes time to generate release notes. sig/planner SIG: Planner size/XL Denotes a PR that changes 500-999 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

reduce the statistics collected for non-predicate columns during ANALYZE

2 participants